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evaluator  training,  and  counseiinz  oroce 


Final  Report:  for  '/ears  One  and  Two: 

Applied  Research  on  Performance  Feedback  and  Apprarsal 

Cynthia  D.  Fisher 
Rcseanne  J.  Fcti 
James  3.  Shaw 

The  objective  of  this  project  ! ccntract  N00014-35-K-0239 )  has 
been  to  provide  developmental  and  basic  research  necessary  to 
support  the  design,  implementation,  and  evaluation  of  the  new 
Marine  Corps  counseling,  goal  setting,  and  feedback  system. 
Research  was  also  conducted  to  aid  the  Marine  Corps  in  the  use 
and  development  of  the  Officer  Fitness  Report  system.  Research 
in  each  of  these  areas  will  be  summarized  separately  below.  In 
both  areas,  recent  advances  in  our  understanding  of  rater 
cognitive  processes  and  ratee  self-assessment  have  been  brought 
to  bear  in  recommending  state-of-the-art  evaluation  devices, 
evaluator  training,  and  counseling  procedures. 

Counseling  System  Related  Research 

We  have  worked  with  MMCE  on  the  development  of  the  new  Marine 
Corps  counseling  system,  both  by  providing  informal  advice  as 
requested,  and  by  preparing  five  technical  reports  relating  to 
performance  counseling.  TR-ONR-8  is  an  extensive  review  of  the 
literature  on  the  performance  appraisal  interview,  focusing  on 
how  feedback  from  the  superior  can  be  conveyed  most  effectively 
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to  subordinates.  TR-GN'R-3  is  a  reporr  of  an  empirical  study  of 
four  methods  of  conveying  feedback  in  a  face-to-face  appraisal 
interview.  This  research  shewed  that  participative  methods  are 
superior  to  a  unilateral,  top-down  approach  to  giving  feedback, 
and  that  the  best  participative  method  seems  to  involve 
subordinate  self-ratings  which  are  explicitly  discussed  in  the 
appraisal  interview.  This  method  assures  that  the  subordinate 
has  thought  about  his  or  her  performance  and  has  become  familiar 
with  the  rating  instrument  prior  to  the  appraisal  interview.  It 
also  assures  that  the  superior  is  made  aware  of  additional 
performance  information  or  points  of  view  which  might  otherwise 
have  been  ever-looked. 

We  believe  that  self  assessment  is  a  crucial  component  of  the 
performance  appraisal  process.  Individuals  have  beliefs  about 
their  own  performance  level,  and  these  beliefs  must  be  recognized 
when  a  superior  tries  to  give  performance  feedback.  TR-ONR-4  is 
an  extensive  review  and  theoretical  development  of  the  process  of 
the  self  assessment  of  performance.  Numerous  reasons  are  given 
for  the  oft-noted  finding  that  self  assessments  disagree  with 
superiors'  assessments  of  performance.  TR-0NR--14  provides  an 
empirical  look  at  self  assessment.  Subjects  were  found  to  be 
quite  inaccurate  when  assessing  their  performance  cn  a  variety  of 
tasks.  The  extent  of  inaccuracy  was  only  weakly  and 
occassionally  related  to  several  personality  variables  which  had 
been  hypothesized  to  affect  the  veracity  of  self  assessments. 


A  final  task  in  connection  with  the  work  for  MMCE  has  been  to 


evaluate  the  new  performance  counseling  program.  A  pre¬ 
implementation  survey  of  attitudes  reward  appraisal,  role 
clarity,  perceived  frequency  and  adequacy  of  feedback,  and  so  on 
was  conducted  in  the  spring  of  1986.  TR-ONR-7  is  based  cn  this 
data  and  explores  some  of  the  causes  and  consequences  of 
superior/subordinate  disagreement  on  performance  rarir.gs.  The 
post-implementation  survey  was  to  have  been  conducted  in  the  fall 
of  1986,  but  due  to  delays  in  implementation  has  had  to  be 
postponed.  This  task  will  be  completed  in  the  third  year  sf  the 
proj  ect . 


Fitness  Report  Related  Research 

Our  second  major  thrust  has  involved  basic  and  applied  research 
on  performance  ratings  from  the  points  of  view  of  instrument 
design  and  other  factors  affecting  rater  accuracy.  This  work  has 
been  conducted  in  conjunction  with  MMPE,  and  in  support  of  a 
proposed  modification  of  the  Officer  Fitness  Report  system.  Over 
the  past  three  years,  we  have  advised  MMPE  on  rating  scale  format 
and  dimensional  content.  It  appears  that  major  changes  in  the 
existing  format  are  not  feasible  at  this  time,  so  our  efforts 
have  turned  to  developing  auxiliary  materials,  such  as  rater 
training  aids,  to  improve  the  quality  of  ratings  made  on  the 
existing  scales.  TR-ONR-2  reviews  the  literature  on  rater 
training. 
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We  are  in  the  process  of  writing  a  report  on  proto- type  anchors 
for  the  performance  dimensions  now  in  use.  Used  in  a  rater 
instruction  manual,  these  anchors  would  clarify  the  meaning  of 
each  dimension  and  each  performance  level,  thus  aiding  raters  in 
using  the  scales  consistently.  The  anchors  are  based  on  three 
waves  of  data  collected  from  Marine  Corps  personnel.  The  rirst 
survey  collected  actual  incidents  of  good,  average,  and  poor 
behavior  on  each  dimension.  The  second  survey  verified  the 
classification  of  these  incidents  into  performance  dimensions, 
and  the  third  survey  scaled  the  items  as  to  perrormance  .evel. 
Supplementary  analyses  of  these  data  will  permit  us  to  assess 
whether  different  performance  schemata  exist  at  different  ranks. 
If  such  differences  are  found,  they  could  account  for  some  of  the 
disagreement  between  superiors  and  subordinates  in  the  evaluation 
of  specific  performances.  TR-ONR-10  applied  a  similar  procedure 
to  the  job  of  patrol  officer  in  two  law  enforcement  agencies. 
Differences  in  the  performance  schemata  of  officers  and  their 
superiors  were  found  to  exist  in  these  samples. 

Additional  research  has  addressed  other  factors  which  may  affect 
the  accuracy  of  performance  ratings  made  by  superiors.  TR-ONR-1 
investigated  the  cognitive  processes  involved  in  perceiving, 
storing,  and  recalling  performance  data  to  make  ratings.  TR-ONR- 
5  addressed  a  methodological  problem  with  much  of  the  laboratory 
research  on  performance  appraisal.  TR-ONR-6  continued  this  line 
of  research  and  showed  that  rating  errors  increased  when  raters 
had  other  tasks  competing  for  their  attention,  a  more  realistic 


scenario  for  actual  ratings  made  in  organizations.  The  authors 
concluded  that  it  is  crucial  for  raters  to  make  a  correct  initial 
categorization  of  ratee  performance,  as  their  sensitivity  re 
subsequent  performance  cues  may  be  low.  It  is  suggested  that 
proto-type  based  frame-of-ref erence  training  might  be  the  best 
method  of  improving  rating  accuracy  by  facilitating  correct 
initial  categorization.  TR-ONR-13  reports  the  results  cf  a  study 
on  rating  accuracy  when  the  superior  is  concurrently  performing  a 
stressful  or  non-stressful  task.  This  study  concluded  that  doing 
a  stressful  task  may  interfer  with  accurate  ratings  by  inhibiting 
recall  of  subordinate  behavior.  Finally,  TR-ONR-15  explored  the 
cognitive  and  motivational  bases  of  leniency  error.  Raters  were 
found  to  be  more  lenient  both  when  they  knew  that  they  would  have 
to  give  face-to-face  feedback  on  their  raring  (motivationally 
induced  bias ) ,  and  when  ratings  were  made  from  memory 
(cognitively  based  bias).  These  results  have  implications  for 
the  design  of  rating  systems  for  the  purpose  of  reducing  leniency 
error. 

Other  Research 

TR-ONR-9  is  a  literature  review  on  the  subject  of  feelings  of 
boredom  in  work  and  life.  This  topic  first  surfaced  in  1981  when 
we  were  interviewing  Marines  at  Camp  Pendleton  as  part  of  a  study 
of  adjustment  to  an  overseas  unit  rotation.  Boredom  both  on  and 
off  duty  was  a  common  complaint,  but  one  seldom  addressed  in  the 
literature  on  management  or  industrial  psychology.  TR-ONR-9 


resents  some  qualitative  data 


reviews  the  existing  literature,  p 
on  the  causes  of  boredom  and  the  actions  people  take  when  they 
feel  bored,  and  suggests  directions  for  further  research  on 
boredom. 


TR-ONR-11  and  TR-CN’R-12  are  a  diversion  into 
field  of  strategic  human  resource  management, 
impact  of  variable  such  as  establishment  size 
technology,  and  unionization  on  a  variety  of 
practices . 


tr.e  new  resear  or. 
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Further  details  on  the  fourteen  technical  reports  released  under 
contract  N00014-35-K-0289  appear  in  Table  1. 
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Table  1 

Technical  Resorts  Issued 
Years’  Cr.e  sr.d  Two 
Contract  ?SCO014-o:-E- 


1|T 


\l\ 


a 


3-ONR-i 


Frank-Majcr.  S  I.,  and  Foti.  R.J.,  (October,  1935). 
Accuracy  in  Performance  Aooraisals:  A  Comparison 
of  Two  Rater  i::n:tive  .-recess  Models. 

ADA  =AI52-355 . 


Two  different  theories  of  the  cocnitive  orocesses 
involved  in  ratine  oerfoniance  were  ccmDared  by 
Nathan  and  Lord  in  1983.  These  theories  comprised 
Borman's  (1978)  traditional  model  of  dimensional 
schemata  and  Reliman's  (1981)  cocnitive 
cateconzaticn  theory.  To  further  explore  the  role 
of  each  in  the  process  of  performance'  aDDraisal  over 
time,  participants  in  the  present  study' were  pre¬ 
sented' with  two  different  videotapes  of  a  lecturing 
college  instructor.  One  half  of  the  participants 
initially  viewed  a  tape  exhibiting  primarily'  good 
lecture  oehaviors  ana  two  days  later  viewed  a 
videotape  containing  mostly  poor  performance 
behaviors.  The  order  of  videotape  presentation 
was  manipulated  so  that  the  remaining  participants 
viewed  the  tapes  in  reverse  order.  subseauently. 
subjects  appraised  the  lecturer's  performance  ch 
a  series  of*  Likert  rating  scales  and  also  com¬ 
pleted  a  Questionnaire  examining  their  ability  to 
recall  specific  performance  behaviors.  These 
measures'  revealed  partial  support  for  both 
theories.  In  general,  the  performance  ratings 
lent  support  to  the  traditional  model  while 
cognitive  categorization  was  corroborated  in  the 
recognition  memory  task. 


TR-ONR-2  DeGregorio,  M.B.  and  Foti,  R.J.  (October,  1985). 
A  Review  of  Rater  Trainina  Approaches  and 
Effectiveness.  ADA  ?I&2-7o6‘.‘ 


The  literature  on  rater  error  reduction  training 
is  first  reviewed  and  assessed.  While  rater  halo 
and  lenienq  error  can  sometimes  be  reduced  by  this 
type  of  training,  rater  accuracy  may  not  be  im¬ 
proved.  Second,  the  literature  on  rater  accuracy 
training  is  reviewed,  and  finally  new  training 
aporoacnes  based  on  rater  cognitive  processes  are 
discussed. 
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TR-CNR-3  DeGrecrono,  M.3. ,  Fisher.  C.D.  and  Fields,  ’!. 

(October.  1585).  Reactions  to  Varying  Forms  c: 
Participation  m  the  .-.ppraisai  interview. 

ABA  5A1o2-:79. 

In  the  laboratory,  subordinate  reactions  to  feedback 
given  in  four  different  wavs  were  assessed.  I'ethcd 
one  was  unilateral,  top-down  feedback,  "ethcd  two 
was  supervisory  feedback  with  the  subordinate 
encour'aced  to  participate  in  the  performance  discus¬ 
sion.  Methods  three  and  four  involved  a  self 
aDDraisai  instrument  completed  Drier  to  the 
participative  performance  discussion.  In  method 
three,  the  seif  anoraisai  did  not  ficure  in  the 
discussion  while  in  method  four  it  was  the  heart  of 
discussion.  All  participative  methods  tended  to 
result  in  more  positive  subordinate  perceptions  than 
the  unilateral  method,  but  no  one  particular  parti¬ 
cipative  technique  was  consistently  the  best.* 


I3-CMS-4  Fisher,  C.D.  and  Russ.  G.  (June.  1586).  Seif  and 
Superior  Assessment.  ADA  #A173-763. 

This  theoretical  paper  beoins  bv  document ina  the 
seventy  ana  pervasiveness  of  disagreements 'between 
superior  and  self  assessments  of  work  performance. 
To* understand  this  disagreement,  theoretical  and 
applied  work  on  self  assessment  is  needed,  similar 
to‘  the  work  on  superior  cognitive  processes  in 
rating  which  has  been  occurring  for  several  years. 
This  paper  draws  on  several  theories  to  explain  the 
self  evaluation  process  and  contrast  it  to’  the  pro¬ 
cess  used  bv  superiors.  A  model  of  seif  evaluation, 
future  research* needs,  and  some  suegestiens  for 
improving  agreement  are  presented.' 


TR-ONR-5  Foti,  R.J.  and  Cohen.  B.A,  (October,  1586). 

Superior  versus  Observer:  Enhancing  the  External 
Validity  of  Laboratory  Performance  Appraisal 
Studiesl 

This  research  evaluated  differences  in  the  psycho¬ 
metric  quality  of  supervisor  vs  observer  performance 
ratings.  Specifically,  type  of  rater  (supervisor 
vs.  observer)  and  type  of  instructions  {ratings  vs. 
neutral  instructions)  were  manipulated  m  a  2  X  2 
factorial  design  to  compare  the  traditional  labor¬ 
atory  performance  appraisal  approach  with  a  more 
realistic  experimental  design.  Results  indicated 


sunerviscrs  csmcnstrstea  r.cre  nsio  =nc  .eniencv 
error  m  rating  iuzcrdmates  behavior  than  did 
observers .  Tvpe  of  mstrucoions  given  ::  :he 
raters  o.ad  no  effect. 


I3-0NR-6  Foti.  R.J.  and  Hauer.soen.  II.M.A.  (October.  1086). 
The  Effects  of  Increasing  Information  Processing 
Demands  cn  Rating  OutccflT  ‘ 

This  research  investigated  the  cognitive  orccesses 
which  mediate  the  oerformance  rating  orccess. 
Soeciiicaiiy,  level  of  processing  ana*  rates  prior 
performance  information' were  manipulated  in  a  3  X  3 
ractoriai  desicn  in  order  to  assess  the  imoact  cn 
osvchcmetnc  rating  outcomes  and  ratine  accuracy . 
Results  indicated  that  as  information  orocessmg 
demands  increased,  raters  relied  more  bn  the  oast 
performance  cues.  Soecificailv,  raters  using' more 
automatic  processing'  and  receiving  a  good  oerfor- 
mance  cue  gave  more  lenient  ratings,  and  those  using 
automatic  processing  and  receiving  a  ooor  oerfor- 
mance  cue  exhibited  increased  halo.  In  addition, 
raters  were  least  accurate  in  recognizing  behaviors 
consistent  with  their  performance  cue.  ImDlications 
for  future  research  in  performance  appraisal  are 
also  discussed. 


TR-ONR-7  Shaw,  J.B.  and  Fisher,  C.D.  (December,  1986). 

SuDervisor-Suborainate  Agreement  on  Performance 


Data  on  several  hypothesized  antecedents  and 
consequences  of  sunordinate  agreement  with  the 
performance  feedback  received  from  their  supervisors 
were  collected  from  over  700  employees.  It  was 
predicted  that  subordinates  who  agreed  with  their 
superior  would  report  that  their  superior  had 
higher  credibility  and  gave  more  freouent,  positive, 
ana  clear  feedback  than  subordinates  who  disagreed 
with  their  superior.  High  agreement  was  also 
hypothesized  to  result  in  higher  effort  to  perfor¬ 
mance  expectancies,  lower  levels  of  role  conflict 
and  ambiguity,  higher  satisfaction  with  the  super¬ 
visor  ana  job,  greater  intent  to  remain  in  the 
organization,  and  greater  commitment  to  the  organi¬ 
zation  than  low  agreement.  Ail  hypotheses  were 
supported. 
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TR-ONR-3  Biddle.  I.  ana  Fisher.  C.D.  (January.  1937). 

Performance  Appraisal  Interview:  A  Review  of 
Research. 

The  research  cn  cerfcrmance  anrrsisai  interviews 
(PAI)  is  reviewed  and  cnncued.  Factors  affect¬ 
ing  superior  and  subordinate  reactions  to  the 
interview  induce  (i)  the  structure  of  the  PAI 
(purccse,  frecuencv,  source  of  appraisal,  and 
ratine  format)';  dl  the  content  of  the  PAI 
(feedback  sicn.  scecificity,  and  performance 
attributions);  (3j  aoai  setting  m  the  PAI:  and 
(4)  the  orocess  or  the  PAI  (participation  level 
and  supervisor  suoportiveness).  Suggestions  for 
designing  effective  PAIs  are  given. 


TR-ONR-3  Fisher,  C.D.  (February.  1987).  Boredom:  Construct, 
Causes  and  Consequences. 

The  literature  cn  causes  of  work  and  life  boredom  is 
reviewed.  A  qualitative  study  of  causes  of  boredom 
cn  and  off  the  job  is  presented,  and  a  tvpolooy  of 
causes  is  develcDed.  Problems  of  measuring  both 
experienced  boredom  and  the  boredom  potential  of 
situations  are  discussed,  followed  by  suogestions 
for  future  research. 


TR-ONR-IO  Foti,  R.J.  and  Hauenstein,  N.M.A.  (Hay,  1987). 

Differences  in  Performance  Schemata  as' a  Function 
of  Organizational  Level. 

The  strategy  of  training  raters  to  adopt  the  same 
evaluative'standard  has  become  a  common  practice  in 
laboratory  performance  appraisal  research.  We  felt 
that  in  the  applied  setting  this  "frame-of- 
reference"  rater  training  strategy  should  be 
expanded  to  include  the  ratees ’  standards  in  order 
to  clarify  workers'  understanding  of  organizational 
expectations.  This  study  explored  the  necessary 
foundations  for  using  this  rater  frame-of-reference 
training  strategy.  A  modified  behaviorally  an¬ 
chored  scaling  method  was  used  to  gather  data  in 
two  law  enforcement  agencies.  The  goals  of  this 
study  were  to  identify  performance  schemata  for  the 
'position  of  patrol  officer,  and  to  assess  how  the 
schemata  differed  by  organizational  level  (i.e. . 
patrol  officers  versus  their  supervisors).  Data 
were  analyzed  using  repeated  measures  analyses  of 
variance  and  discriminant  analyses.  Differences 
in  performance  schemata  between  organizational 
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levels  were  tentatively  identified.  Findings  were 
discussed  in  relation  to  the  needs  ci  the  two 
agencies  and  in  terms  of  general  inclinations  for 
rater  training  strateaies. 


MR-11  Fisher.  C.D.  ana  Shaw,  J.3.  (April,  1937).  Ices 
Business  itrateoy  Affect  Personnel  Mansaement 
Practices/  ‘ - 

The  effect  of  business  strategy  (defender  versus 
analyzer  and  orcscectcr!  on  a'vanety  :f  human 
resource  management  cractices  was  assessed. 
Hypotheses  regarding  selection,  training  and 
ccmper.saticn  were  supported.  Those  regarding  job 
analysis,  use  of  written  HRI1  nolicies/crcmoticn 
practices,  and  oerfontance  anbraisal  were  net 
suppertea.  Factors  which  may  weaken  the  stratecy- 
practice  match  were  discussed  extensively. 


MR-12  Fisher.  C.D.  and  Shaw,  J.B.  (May,  1987) . 
Determinants  of  Personnel  Practices. 

There  has  been  relatively  little  research  aimed 
at  identifying  the  factors  which  are  associated 
with  the  use  of  sophisticated  and  bureaucratic 
personnel  management  methods  as  oocosed  to  casual 
and  informal  methods.  Existing  theory  and  research 
suggest  that  headquarters  location,  establishment 
size,  technology,  extent  of  unionization,  ar.d  the 
existence  of  a  designated  oersonnel  decartment 
may  be  among  the  causal  factors.  The  relative 
impact  of  these  factors  on  the  soohisticaticn  of 
recruiting,  selection,  comoensaticn  and  cer- 
lormance  aopraisal  was  assessed  in  a  sample  of 
174  establishments. 


TR-QNR-13  Powell,  A.B.,  Shaw,  J.B.  and  Fisher,  C.D.  (May, 
1987).  The  Effects  of  Rater  Stress  on  Performance 
Rating  Accuracy. - - - 

This  study  examined  the  effects  of  rater  stress  on 
halo  error,  severity  error  and  rating  accuracy  in 
performance , aopraisal.  Eiahty-four  participants 
completed  either  a  stressful  or  nonstressful  in- 
basKet  exercise,  either  before  or  after  observing 
the  videotaped  performance  of  a  manager.  They  then 
rated  the  manager  on  several  performance  dimensions 
Ratings  provided  by  participants  completing  the 
stressful  in-basket  were  less  accurate  than  those 
provided  by  participants  completing  the 
nonstressful  in-basket.  However,  ratings  provided 
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37  these  two  crcuos  shewed  no  differences  in  halo  :: 
sever::?  error.  In  addition,  raters  who  completed 
the  in-cesket  otter  seemc  the  tercet  performance 
but  before  rating  that  oef finance' provided  less 
accurate  ratines "than  those  completing  the  in- 
basket  oef  ore  ebservne  and  ratine  the  performance. 
These  results  indicate" that  rater'stress  :,av  lead 
to  creater  aooraisal  inaccuracy,  and  that  this 
inaccuracy  is* largely  a  function  of  faulty 
information  retrieval. 


Fisher.  1.2.  and  Chachere,  G.  (May, 
The  Self-Assessment  of  Performance: 


1937). 

Three  Studies. 


Possible  determinants  of  the  accuracy  of  seif- 
ratincs  cf  oerfcrmance  are  discussed!  The 
reiatienshibs  of  oast  experience  on  the  task, 
locus  of  control," and  dispositional  private  and 
public  seif-focus  to  self* ratines  are  explored  in 
two  correlational  studies  usmg'different  tasks. 
Private  seif-focus  sometimes  results  m  iess 
accurate  self-rating,  contrary  to  the  hypothesis. 

A  third  study  manipulates  seif-focus  situaticnaily 
but  finds  no  effect  on  self-rating  accuracy. 
Overall,  it  is  concluded  that  the  process  of  self- 
assessment  is  not  well  understood. 


Hauenstein.  Neil  M.A.,  Whitcomb,  Alan  J.,  and 
Foti.  Roseanne  J.  (Auoust,  1387), 

Effects  of  Future  Interactions  and  Time  Delays  on 


This  study  tested  three  potential  explanations  cf 
evaluator  leniency.  The* first  explanation  viewed 
leniency  as  a  motivational  bias  affecting  the 
recorded  rating,  but  not  the  rater's  actual 
impression.  The  second  viewed  leniency  as  a 
cognitive  retrieval  bias  induced  by  demands  uccn 
memory.  The  final  explanation  viewed  leniency  as 
the  interplay  of  motivational  and  coonitive 
processes  causing  an  encoding  bias  which  in  turn 
resulted  in  lenient  ratings.  These  three 
explanations  were  tested  in  a  laboratory 
study, in  which  raters  provided  both 
immediate  and  memory  based  ratings.  Raters' 
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motivation  was  nanmuistea  tnrcucn  a 
between  subiects  factor  which  varied  exnect- 
aticns  cf  crcvidmu  face-to-face  feedback  :c 
the  rates.'  Succor:  was  found  for  the  native::: 
and  cognitive  retrieval  biases,  cut  contrarv  :: 
excectaticns  nc  succor:  was  found  far  the  ehccc 
bias  excianaticn.  ‘ P.esults  were  discussed 
relation  to  advancma  the  understandir.a  o:  the 
performance  aonraisal  orccess  throucn  the 
incorncration" of  motivational  factors. 
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